282

|

6 Applications and Methods in Biosignal Processing

These coefficients Hi were taken as features to evaluate the performance of several

classification algorithms, however the best results were achieved using a simple K-

nearest neighbour algorithm (KNN) with K=10. The KNN algorithm is a non-parametric

supervised learning method; it was first defined by Evelyn Fix and Joseph Hodges in

1951 [16] and later expanded to regression methods by [1]. An object is classified by

a plurality vote of its neighbours in the feature space, with the object being assigned

to the class most common among its K nearest neighbours (K is a positive integer,

typically small). If K = 1, then the object is simply assigned to the class of that single

nearest neighbour.

As explained in [25], the euclidean distance is used as metric between the data

points. It is well known that the performance of the classifier depends on the correct

choice of K. If K is small it may happen that noise in the training sets decreases the

accuracy. If K is chosen too big, it is possible that distant points influence the classi-

fication decision. In our case, we choose K = 10 and use the classifier as implemented

in Matlab (see Listing 6.3.4).

Obviously, the amount of data in the clinical study is small. To deal with that prob-

lem, a procedure described in Figure 6.48 is employed and histograms of the resulting

accuracies are created.

Step 1 (correct labels):

Do N-times

– Create a new random M-fold cross validation

– Calculate the accuracy

Step 2 (comparison with incorrect labels):

Do NR-times

– Randomly permute the labels

– Do N-times

– Create a new random M-fold cross validation

– Calculate the accuracy

Fig. 6.48: Procedure to test whether the performance of a classifier trained and tested on correct

labels exceeds the performance of a classifier trained and tested on incorrect labels. For the results

in this paper, we always choose N = 100, NR = 100 and M = 10. Algorithm is quoted from [25,

Figure 3].

We quote from [25] that this was motivated by large deviations in the achievable ac-

curacies depending on the split in training and test set. Further, the comparison with

the achievable accuracies on classifiers trained with randomly permuted labels gives

us an indicator if the used method is prone to overfitting due to hyperparameter tun-

ing. Whilst it is evident that the cross-validation could have been left out in this case,

by instead creating only one training and test set in the loop, this procedure gives a